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Summary 


This report presents the Education Policy Institute and Renaissance Learning's first 
assessment of the learning loss experienced by pupils in England as a result of the 
coronavirus (COVID-19) pandemic. 


The analysis is based on the results achieved by pupils in the first half of the 2020/21 
autumn term (up to and including 25 October 2020) in comparison to pupils in previous 
years. Star Assessments are frequently used by schools as their baseline assessment for 
reading and maths, and as such the highest number of assessments are conducted 
during the autumn term. 


In Part 1 we examine the achieved sample sizes. We find that: 


Just over 1 million assessments were carried out in the first half of the autumn 
term, the majority of which were in reading. 


The number of assessments in primary schools increased this year (by around 14 
per cent), however there has been a drop in secondary schools (of around 23 per 
cent). The number of assessments is still higher in secondary than in primary 
schools. 


The achieved sample sizes mean that analysis is focussed on year groups 3 to 9. 


In Part 2 we examine historic trends in Star Assessments to develop an approach to 
measuring learning loss. We find that: 


The progress that pupils make in terms of scaled scores changes as pupils get 
older. The difference in mean scores between one year group and the next is 
greater in primary schools than it is in secondary schools. In other words, the rate 
at which the average pupil improves in terms of scaled scores from one year to the 
next falls as they get older. 


Consistent with other assessments, progress is not consistent within school years. 
Pupils appear to make more progress at the start of the academic year, which 
then begins to flatten off later in the year, at some points results are actually lower 
than in previous assessments. 


To a certain extent, results in the autumn term of any year are likely to be subject 
to the effects of normal summer learning loss. In a limited number of instances, 
mean attainment in the autumn term was lower than it had been in the previous 
summer. Assessments of learning loss need to account for this. 


We present a method for estimating learning loss that is based on the expected 
progress for pupils based on their prior attainment and historic rates of progress 
for similar pupils. 


In Part 3 we estimate the mean learning loss in reading (for primary and secondary 
pupils) and mathematics (for primary pupils only due to small sample sizes in secondary). 
We find that: 


e All year groups have experienced a learning loss in reading. In primary schools 
these were typically between 1.7 and 2.0 months, and in year 8 and year 9, 1.6 
and 2.0 months respectively. 


e tis pupils who have just started year 7 where we see the smallest learning loss at 
0.9 months. One possible explanation for this is that this year group was out of 
school for less time than others. Schools were expected to be open to reception, 
year 1, year 2 and year 6 from 1 June 2020 — while others were still receiving 
remote education. Though it should be noted that the sample size for this group is 
lower than other year groups and these results may be adjusted with more 
detailed data available later in the year. 


e The learning losses in mathematics were greater. We estimate that, on average, 
pupils in primary schools have experienced a learning loss of just over three 
months. It has not been possible to derive robust estimates for pupils in secondary 
school in mathematics. 


e There appear to be some regional disparities in the level of learning loss in 
reading with pupils in the North East and in Yorkshire and the Humber seeing the 
greatest losses. However, the differences between regions are relatively small 
once we control for historic rates of progress in these areas and all regions have 
experienced losses. 


e We also find schools with high levels of disadvantage have experienced higher 
levels of loss than other schools, particularly in secondary (2.2 months in schools 
with high rates of free school meal eligibility and 1.5 months in schools with low 
rates of free school meal eligibility). 


e At this stage it is not possible to break down results by pupil characteristics, or to 
model the progress typically made in Star Assessments by different pupil groups, 
but this will be possible when the data are matched with information held in the 
National Pupil Database and will be included in our full report. 


Background 


This is the first report from a collaborative and innovative research project carried out by 
Renaissance Learning UK Ltd (Renaissance) and the Education Policy Institute (EPI) on 
behalf of the Department for Education. The aim of the project is to measure the impact 

of coronavirus (COVID-19) on the academic development of children and young people 

in England. 


In autumn 2020, over a million tests in Renaissance Learning’s Star Reading and Maths 
were sat by pupils in England. These assessments employ computer adaptive tests that 
continually adjust the difficulty of a test, selecting assessment items based on a student’s 
performance on each previously answered item. The Star Reading assessment 
measures students’ performance on key reading skills via a brief standards-based test of 
general reading achievement, administering 34 questions that students complete, on 
average, in less than 20 minutes. The Star Maths assessment similarly comprises a brief 
assessment of 34 questions that students complete, on average, in less than 25 minutes. 
Both draw on item banks of over 6,000 items. 


The aim of this report is to use Renaissance’s historical Star Assessment data alongside 
these tests to provide an estimate of the level of learning loss as a result of schools 
‘closing’ for the majority of pupils in the summer term of 2019/20 with an associated 
move to remote learning. 


There have already been several studies that have attempted to estimate the extent of 
learning loss as a result of the pandemic and the effect of the pandemic on pupils from 
disadvantaged backgrounds. 


In October 2020, the organisation No More Marking published their analysis of a writing 
exercise carried out by over 116,000 year 7 students at the start of the academic year. 
Through an assessment of pupils’ work they concluded that the work pupils produced 
was similar to that of year 5 pupils from an exercise in November 2019.° In other words, 
they concluded, students were 22 months behind where they expected them to be. 
However, this analysis comes with important caveats. First, that year 7 students had not 
previously undertaken this exercise. Secondly, the exercise was carried out in the first 
half of the autumn term whereas previous assessments had been in the second half of 
the autumn term. In other words, there was no direct comparison with similar pupils in the 
past. It is also likely, as their analysis sets out, that the degree of learning loss will vary 


1 A more detailed discussion of Star assessments is available in ‘Research Foundation for Star Adaptive 
Assessments — Science of Star’, Renaissance White Paper, September 2020. 

? The assessments used in this report were carried out in school. As remote learning has continued into the 
spring term of 2020/21, Renaissance Learning has encouraged the use of assessments remotely and 
these may be used in future reports. Evidence suggests that assessments are consistent between the two 
approaches except for the very youngest children 

3D. Christodoulou, ‘Baseline secondary writing: have year 7 pupils gone backwards?’, October 2020. 
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by subject. For example, estimates of learning loss in reading may be lower since it is 
more likely that pupils will be reading regularly at home than undertaking written 
exercises. 


In November 2020, RS Assessment (Hodder Education) published an analysis of tests 
taken by 250,000 primary school pupils in the autumn of 2020.4 These tests included 
mathematics, reading, and grammar, punctuation, and spelling. They identified significant 
reductions in attainment which varied by levels of disadvantage and the age of pupils, 
though these were not translated into months of lost learning. These tests are normally 
carried out in the summer and the report sets out that differences will be caused by a 
combination of higher pupil age, the replacement of in-person teaching with remote 
learning, and home schooling, and the effects of learning loss during the school summer 
holiday. 


Even with these limitations, it would suggest that pupils from different backgrounds have 
been affected by the pandemic in different ways. The Education Endowment Foundation 
(EEF) estimated that the disadvantage gap could widen by between 11 and 75 per cent 
and reverse progress in closing the gap since 2011.° However, as their report makes 
clear, there are several limitations to this analysis. Primarily that the school closures 
considered in the evidence review were somewhat different from those experienced in 
England in 2020. Subsequently, in analysis based on assessments taken in the autumn 
term of 2020/21 by 6,000 Year 2 pupils, the EEF estimated overall learning loss in 
reading and mathematics of around two months when compared to a group of Year 2 
pupils assessed in 2017.° They also presented initial estimates of the effect on the 
disadvantage gap which suggest it may have widened but at the lower end of earlier 
estimates. 


The key strength of the Renaissance assessment data, and hence this report, is that it 
allows us to link outcomes in the first half of the autumn term 2020/21 with outcomes at 
the same point in 2019/20 at pupil level and compare their progress with similar pupils in 
earlier academic years. The samples are also sufficiently large enough to allow us to see 
whether these results are consistent across different year groups. An analysis of 
Renaissance assessment data has already been carried out for students in the United 
States.’ It found small negative impacts in reading and moderate impacts in 
mathematics. These varied by year group with some year groups up to 7 weeks behind in 


4K. Blainey and T. Hannay, ‘The impact of lockdown on children’s education: a nationwide analysis’, 
November 2020. 

° EEF, ‘Impact of school closures on the attainment gap: rapid evidence assessment’, June 2020. 

5 EEF, ‘Impact of school closures and subsequent support strategies on attainment and socio-emotional 
wellbeing in Key Stage 1: Interim Paper 1’, January 2021 

7 Renaissance, ‘How kids are performing — tracking the impact of COVID-19 on reading and mathematics 
achievement’, October 2020. 
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reading and some year groups up to 12 weeks behind in mathematics. Students 
attending schools serving low-income families were also more negatively affected. 


As well as headline findings for pupils in England, this report also includes some analysis 
by school level demographic data. Subsequent reports from the EPI/Renaissance project 
will incorporate data that have been linked with the National Pupil Database to enable 
analysis to carry out detailed analysis by pupil characteristics (such as pupils from 
disadvantaged backgrounds). It will also include further assessments taken during the 
2020/21 academic year. This will enable us to assess the extent to which pupils who 
experienced a learning loss then catch-up. 
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Part 1: Assessments, data and sample sizes 


Data is drawn from assessment data from Renaissance Learning’s Star Reading and 
Star Maths, which are used by over 3,000 primary schools and nearly 2,000 secondary or 
all-through schools in England. The assessments provide criterion-based scores that run 
on a singular scale from year 1 to year 13. 


Renaissance Learning provided a dataset to the Education Policy Institute comprised of 
all assessments undertaken in schools in England between 1 August 2017 and 25 
October 2021. A total of 11,685,315 records were provided (one per assessment), a full 
list of variables is provided in the Annex. 


Figure 1.1 shows the patterns in the number of assessments undertaken in each of 
reading and mathematics by half-term from autumn 2017 onwards, a full record of the 
number of assessments is provided in the Annex. 


The number of assessments undertaken in both reading and mathematics has been 
growing rapidly in recent years. In the first half-term of the 2017/18 academic year, just 
over 800,000 assessments in reading were undertaken; by the same point in the 2019/20 
year this had grown to over 1.1 million. The number of assessments is typically much 
higher in the autumn than in the spring and summer terms as schools use them to 
provide baseline assessments at the start of the year. But the number during the spring 
and summer terms is still significant, reaching nearly 700,000 at the start of the spring 
term in 2019/20. 


The number of assessments in mathematics has historically been much lower, though 
this figure is growing. At the start of the 2017/18 academic year there were 32,000 
assessments in mathematics and by 2019/20 this had grown to around 55,000. 


There was a slight drop in the number of assessments in 2020/21 with just over a million 
reading assessments taken, though the number in mathematics has held steady. There 
is some indication that schools delayed participation at the very start of the term before 
then returning to more normal numbers. 
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Figure 1.1: The number of assessments undertaken in Star Reading and Star Maths 
by half-term 2017/18 to 2020/21 
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Season term 


The analysis above shows the total number of assessments taken. However, it is 
possible for pupils to take the assessments more than once; around 10% of pupils take 
more than one assessment in a half term. Figure 1.2 shows the number of individual 
pupils who undertook assessments in the first half of the autumn term for each of 
2018/19, 2019/20 and 2020/21 broken down by year group. It shows that: 


e Prior to 2020/21 the number of pupils taking assessments was increasing in all 
year groups. 


e The largest numbers are seen in the first years of secondary school, in 2019/20 
over 230,000 pupils took assessments in reading. 


e In primary schools, the number of pupils taking assessments was higher in 
2020/21 than it was in previous years. Overall, the number of pupils in years 3-6 


taking assessments in reading grew by 14 per cent between 2019/20 and 
2020/21. 


e In secondary schools, the number of pupils taking assessments was lower in 
2020/21 than it was in previous years. Overall, the number of pupils in years 7-9 
taking assessments in reading fell by 23 per cent between 2019/20 and 2020/21. 


e The number of pupils taking maths assessments in 2020/21 grew slightly but 
remains much lower than the number taking assessments in reading. 
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For much of the analysis presented here we are interested not only in the scores that 
pupils achieved this year but also what they have achieved in the past so we can 
measure relative progress. Therefore, in Figure 1.3 we show the number of pupils in 
each autumn first half-term who also had an assessment at the equivalent point in the 
previous academic year (which we will refer to as ‘prior attainment’). 


Figure 1.2: Pupils undertaking assessments in reading and in mathematics in the 
first half of the autumn term by year group — 2018/19 to 2020/21 
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Figure 1.3: Pupils undertaking assessments in reading and in mathematics in the 
first half of the autumn term who also undertook assessments at the same point in 
the previous year by year group — 2018/19 to 2020/21 
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Again, we can see that the number of pupils in primary schools that meet this condition 
increased in 2020/21, though there were falls in the number in secondary schools. Note 
the significant difference between the overall number of pupils in year 7 and the number 
that also have prior attainment. This is likely to be because this is a transition point for the 
majority of pupils and the data linking method employed was unable to identify pupils if 
they had moved school (so would only be included if, for example, they are in an all- 
through or middle school). In subsequent reports we will use a wider set of pupil 
identifiers through the National Pupil Database to obtain larger sample sizes. 


Similarly, the number of pupils in years 1 and 2 is very low. The number of pupils in years 
10, 11 and 12 (combined) is of a similar order to that in year 7. However, the nature of 
participation in Star Reading is such that these pupils may not be representative of pupils 
as a whole — these pupils tend to have lower attainment. 


Taking all of these findings combined, the majority of the analysis in this report focuses 
on outcomes in years 3 to 9. For mathematics we group pupils into primary and 
secondary year groups. Note that when considering learning loss, we include year 7 
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within primary as we are chiefly interested in the learning loss that occurred during the 
summer term when these pupils were in year 6. 
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Part 2: Developing an approach to measuring learning 
loss 


Patterns of progress in Star Assessments during time in school 


Outcomes in Star Assessments are based on a continuous scaled score where pupil 
scores increase as they move through the school system. Pupils in year 3 taking an 
assessment at the start of the academic year typically achieve around 250 points on this 


scale. By the final year of primary school (year 6) this increases to around 550, and by 
year 9 to around 750. 


Figure 2.1 shows the historic mean scores in Star Reading for each cohort as they have 
progressed through different years of schooling. For example, the top line on this chart 
shows the group of pupils who started year 9 in September 2020/21. The very left of the 
chart shows the mean score for this cohort in September 2017/18, when they were in 
year 6, and the very right shows their outcomes in the spring term of 2019/20, when they 


were in year 8 and just as the country entered a national lockdown and the majority of 
pupils received remote learning. 


Figure 2.1: Historic outcomes for the 2020/21 cohort up to and including spring 
term 2019/20 
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Note that in this illustration we have taken the average of all pupils who took 
assessments at that point and therefore it is not necessarily the exact same group of 
pupils taking the assessments at each stage. However, these results suggest two things: 


e Progress is not consistent within school years. Pupils appear to make more 
progress at the start of the academic year which then begins to flatten off later in 
the year, at some points results are actually lower than in previous assessments. 


e Progress is not consistent between school years. The difference in mean 
scores between year groups is greater in primary schools than it is in secondary 
schools. In other words, the rate at which we expect pupils to improve in terms of 
scaled scores from one year to the next falls as they get older. 


This pattern of results is consistent with other studies of pupil progress, including studies 
of pupil progress in England, and is not a particular feature of Star Assessment.® 


Patterns of progress from term to term 


We now consider progress within each academic year further. Rather than looking at 
average attainment across all pupils we can compare the outcomes of pupils who have 
results in two consecutive half terms and measure the average progress that they make 
between those points. 


In Figure 2.2 we look at the mean change in scores in reading between each half-term for 
pupils in year 5 and in year 7. In Figure 2.3 we do the same for mathematics for pupils in 
years 3-6 and 7-9 (due to smaller sample sizes). We see that: 


e the progress that pupils make between different half-terms generally diminishes 
through the year, this is particularly true for younger pupils; 


e the largest increases are seen between the first and second halves of the autumn 
term, these are typically three times the size of the increases seen between the 
two halves of the summer term; 


e the smallest increases are seen between the second half of the summer term and 
the subsequent autumn term. In fact, in mathematics we see a small fall in 
outcomes between these two points. 


8 Department for Education, ‘How do pupils progress during Key Stage 2 and 3?’, March 2011. 
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Figure 2.2: Change in outcomes in reading between half-terms for pupils in year 5 


and year 7 


15 


Ses © OULCS 
i ic 
| 
N 
a. 
rr 
oO 
“0 oe | OCS 
2 oS 5s 
N a ees] 
, 
| 
N 
ls 
ee eesessSsSsS 
oO 
S 
S pues | OOS 
N 
9 cic 
Sas 
N 
CE 
N a a eee al 
eT 
—s—S—sS 
o 
(o>) lo [o) LO oO ike) fo) ike) fo) 
wt oO oO N N - - 


SUUJ9} JJEY UBEMjEq B10NS pejeos Ul eGueUD 


2019/20 


2018/19 


Season term 


2017/18 


m Reading Year 7 


mReading Year 5 


Figure 2.3: Change in outcomes in mathematics between half-terms for pupils in 


primary and secondary schools 
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These results suggest some degree of learning loss over the summer. This does not 
necessarily mean that learning has been truly ‘lost’, pupils may simply be out of practice 
with the tasks being assessed, more generally not academically engaged over the 
summer, and their performance then improves rapidly once they have been back in 
school for a while (and this may explain why the differences are more pronounced in 
mathematics than reading, which is perhaps more likely to continue at home). 


This means that any assessment of learning loss based on assessments early in the 
autumn term that then draw direct comparisons with other points in the academic year 
risk overestimating the learning loss due to the pandemic. 


This leads us to our first two principles for our measurement of learning loss: 


e we should not make direct comparisons between different points of the academic 
year without controlling for the usual differences that occur between them (the 
simplest way, as we use here, is to compare between the same point within each 
academic year — i.e. the autumn term). 


e ins far as this is practical given sample sizes, analysis should be conducted 
separately by year group to allow for differential rates of progress by the age of the 
pupil (or at the very least, grouped with similar ages). 


Changes in mean attainment in 2020/21 


Our first analysis of outcomes in 2020/21 considers the average outcomes in Star 
Reading and Star Maths in the first half of the autumn term in each of the last three years 
by year group (Figure 2.4). 


When we compare results in 2020/21 with those in previous years, we find that: 
e In primary year groups the average results in reading in 2020/21 are lower than in 
both 2018/19 and 2019/20 — by between 10 and 12 scale points. 


e In secondary year groups the average results in reading in 2020/21 are broadly 
the same as in 2019/20 and for year 9 pupils have increased slightly. 


e In mathematics, results in 2020/21 are substantially lower than in 2019/20 for both 
primary and secondary pupils. For primary pupils, results are 23 scale points lower 
than in 2019/20 and for secondary pupils, results are 35 points lower than in 
2019/20 (though note that the sample sizes here are relatively small). 
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Figure 2.4: Mean scores in reading and mathematics in the first half of the autumn 
term 2018/19 — 2020/21 
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These headline results suggest some degree of learning loss in reading that is more 
pronounced in younger than in older age groups (where results have remained steady). 
There appear to be larger learning losses in mathematics, these are observed in both 
primary and secondary aged pupils with older pupils seeing larger falls. 


But these results also have two key limitations. Firstly, they do not account for the fact 
that the number of pupils taking these assessments has been growing, and the profile of 
pupils (in terms of average rates of progress) may be changing. Secondly, they do not 
account for the fact that outcomes have been improving slightly. If, for example we 
consider this year’s year 6 pupils. As a cohort they achieved an average score of 459 at 
the start of year 5, 6 points higher than the cohort that preceded them. 


By comparing results this year with results last year, we make an assumption that they 
should be the same. However, these two limitations suggest that this is not necessarily 
the case. We therefore introduce a third principle into our measurement of learning loss: 


e we should consider how we might have expected results to differ this year 
because of changes to the pupil population. 
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Methodology for estimating expected progress and learning loss 


In this section we introduce an approach to measuring learning loss that builds on these 
three principles. We calculate an expected outcome for pupils this year based on what 
they achieved last year and applying historic rates of progress for pupils with similar prior 
attainment. 


Step 1: We construct a model of the relationship between outcomes and prior 
attainment using historic data: using data from 2018/19 and 2019/20 we examine the 
relationship between what pupils achieved in the autumn of 2018/19 and what they went 
on to achieve in 2019/20. We use a regression model to fit a statistical line of best fit 
through the data. We do this for each year group in each subject (see figures 2.6 and 
2.7). 


Step 2: We then use the model to calculate an “expected outcome” for each pupil 
based on their prior attainment: We use the derived regression line to calculate the 
‘expected’ outcome for a pupil in autumn 2020 if they followed similar patterns of 
progress as in previous years. 


Step 3: Given that we now have a known prior attainment and an expected 
outcome, we can calculate an expected progress and actual progress: as the same 
scale is used throughout, the expected progress is simply the expected outcome minus 
the prior attainment score. Actual progress is simply the actual outcome minus the prior 
attainment score. 


Expected progress = expected outcome — prior attainment 
Actual progress = actual outcome — prior attainment 


The principle of this calculation is the same as that of school performance tables ‘value 
added’ measures such as Progress 8. We compare the outcomes achieved to the 
outcomes of those with similar prior attainment. 


Step 4: We can now calculate the estimate of learning loss as the difference 
between expected progress and actual progress: in terms of scale points this is 
simply subtracting one number from the other. 


Learning loss in scale points = actual progress — expected progress 


We can also convert this into months of learning. Since we are considering the progress 
from one year to the next this is the expected progress over a 12-month period. 
Therefore, the learning loss in months is given by: 


actual progress — expected progress 
Learning loss in months = + X 12 
expected progress 
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Note that such an approach does not provide a strong prediction for an individual 
(typically the prior attainment of a pupil explains between 50 and 60 per cent of the 
variation in outcomes) but becomes more reliable when taken in aggregate.? We 
calculate regression lines for each year group and for each subject. This is to allow for 
any different rates of progress in different year groups from otherwise similar starting 
points. 1° 


Example 


In this example, a pupil has prior attainment of 600 points in 2018/19 and scores 675 
points in 2019/20. 


Step 1: We construct a model of the relationship between outcomes and prior attainment 
using historic data which for this pupil is shown in figure 2.5. 


Step 2: Using this model we find that the expected outcome for this pupil is 700 points. 


Figure 2.5: Example regression line used to generate expected scores and hence 
expected progress" 
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Step 3: Using the expected outcome and prior attainment scores we can calculate the 
expected and actual progress 


Expected progress = 700 — 600 = 100 points 


Actual progress = 675 — 600 = 75 points 


° Note that this explanatory power is similar to that of the model used in Progress 8. 

1° Prior to calculating the reading models, we remove the top and bottom 5 per cent of the prior attainment 
distribution to minimise the effect of extreme values. In mathematics we remove the top and bottom 10 per 
cent. 

11 Note the data presented in this example have been randomly generated due to privacy concerns. They 
do not represent actual scores achieved by pupils who undertook assessments. 
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Step 4: Using the expected and actual progress scores we can calculate the learning 
loss in scale points and in months. 


Learning loss in scale points = 75 — 100 = 25 points 


; 75 — 100 
Learning loss in months = Too 12 = 3 months 


Therefore, in this example the pupil has experienced learning loss of 25 points which is 
equivalent to 3 months of learning. 
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Figure 2.6: Regression lines used to generate expected scores and hence 
expected progress for each year group — reading 
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Figure 2.7: Regression lines used to generate expected scores and hence 
expected progress for each year group — mathematics 
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A full set of regression coefficients is provided in the Annex. 


An aspect we do not control for is the precise length of time between tests. Our 
calculation assumes exactly one year passes between tests, but pupils may have taken 
the test at the start of the half term in one year and the end of the half term in the other. 
This would be particularly problematic if this pattern changed from one year to the next. 
However, in the Annex we present analysis of the distribution of the number of days 
between tests by year group and find little difference between this year and last. We also 
find that the median length of time between tests is broadly one year for all year 
groups. '* As we refine our models in subsequent reports — including adding a range of 
contextual factors to the models — we will assess whether including this as a factor 
improves the reliability of our estimates. 


Having calculated a mean learning loss (or indeed gain) for each individual pupil we are 
then able to aggregate up to our groups of interest. For this initial report this is largely the 
pupil’s year group, but in subsequent reports we will be able to consider a range of pupil 


12 At the extreme, the median length of time was one year plus 6 days. 
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characteristics such as deprivation and ethnic group. We have concluded that the sample 
sizes for secondary mathematics are too small to provide robust estimates at this stage. 


Throughout this report, when we say ‘learning loss’ what we really mean is loss in 
comparison to a situation where pupils make the same progress as similar pupils did last 
year. But the true ‘counterfactual’ may have differed from this. For example, results in 
2019/20 were generally higher than they were in 2018/19. It is possible that, in the 
absence of the coronavirus (COVID-19) pandemic, results in 2020/21 may have been 
higher still. Alongside the results we provide an illustration of how this may affect our 
results and may be incorporated into future modelling. 
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Part 3: Estimated learning loss 


Estimates of learning loss in reading and mathematics 


Having calculated estimated learning loss for individual pupils we now aggregate up to 
pupil year groups. 


In Figure 3.1 we present the modelled learning loss of pupils in Star Reading in years 3- 
9. We find that all year groups appear to have experienced a learning loss and these 
were of a similar magnitude, of around 2 months, across ages — i.e. there does not 
appear to be any real difference between primary-aged and secondary-aged pupils. 
Pupils who have just started in year 6 and pupils who have just started year 9 
experienced the largest losses (2.0 months) but this is not materially greater than those 
in years 3, 4, 5, and 8. 


It is pupils who have just started year 7 where we see the smallest learning loss of 0.9 
months. One possible explanation is that this year group was out of school for less time 
than others. Schools were expected to be open to reception, year 1, year 2, and year 6 
from 1 June 2020 — while others were still receiving remote education. 


It should also be noted that this is the smallest group for which we are presenting results 
and the number of pupils included is significantly lower than in last year’s data. It is 
therefore possible that there is a systematic bias in the schools that have participated in 
Star Reading assessments to this point (e.g. schools that have historically demonstrated 
lower results of progress or pupil groups that make less progress on average). We will be 
able to assess this as more data becomes available, including pupil characteristics data, 
and we will include further analysis in our second report. 


As we set out above, the learning losses presented here are essentially relative to what 
the pupils would have achieved had they followed the same pattern of progress as pupils 
last year. But it is possible for these relationships to change over time — for example if 
standards improve, the characteristics of the cohort vary considerably, or schools that are 
particularly high or low performing leave or join the programme of assessments. 


To illustrate the potential size of these year-on-year fluctuations we apply the same 
approach to historic data (in other words, calculating similar measures for 2019/20 using 
progress data from 2018/19). These are presented in Figure 3.2. Pupils in 2019/20 
typically made more progress than pupils in 2018/19, equivalent to between 0.4 and 0.8 
months of learning. 


It is not possible to say exactly what would have happened in the absence of the 
coronavirus (COVID-19) pandemic and at this stage we believe an assumption of similar 
rates of progress as last year represents the best approach. However: 
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e lf the system had continued to improve at the same rate as between 2018/19 and 
2019/20 then the degree of learning loss is potentially higher — 0.4 to 0.8 months. 


e If 2019/20 was in fact a high-point, then the learning loss is potentially lower — 
again between 0.4 and 0.8 months. 


We will continue to investigate these differences as more data becomes available, 
particularly as we gain access to pupil characteristics data which will enable us to better 
control for differences between pupil cohorts. 


Figure 3.1: Estimated learning loss in reading 2020/21 


Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 
0.0 


-0.5 
-1.0 -0.9 
2020/21 
-1.5 
-1.6 
-1.8 -1.8 
-1.9 
-2.0 -2.0 


Estimated Learning Loss 
(months, negative values indicate loss) 


-2.0 


-2.5 
Year Group 


29 


Figure 3.2: Estimated learning loss in reading 2020/21 in comparison to estimates 
made in a similar way for outcomes in 2019/20 
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The number of cases in mathematics is significantly lower than in reading. As shown in 
Figure 1.3 they amount to 19,000 cases in primary school and just over 1,000 in 
secondary. This means that in primary there is some degree of uncertainty around 
estimates in mathematics and the results are sensitive to the exact specification of the 
model used and the specific circumstances of the pupils included. We expect to be able 
to improve the robustness of these estimates as more data becomes available — for 
example, when we are able to link outcomes to pupil characteristics we will be able to 
better control for any systematic differences between cohorts and the schools that are 
included in the analysis. 


Unfortunately, we have concluded that the sample sizes in mathematics for secondary 
schools are simply too small to draw conclusions about learning loss in mathematics at 
this stage. The numbers mean that any estimate will be based on a very small number of 
schools and any results are likely to reflect the individual circumstances of those schools 
rather than being an estimate of the effect on the system. We will review this position as 
more data becomes available later in the autumn term. 


In Figure 2.4 we showed that there was a substantial reduction in the overall attainment 
in mathematics in primary schools. In Figure 3.3 we present the modelled learning loss of 
pupils in Star Maths for these pupils. 
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The effects are larger than we saw for reading. Amongst primary-aged pupils (including 
pupils in year 7, since the learning loss was primarily experienced during year 6) we 
estimate that pupils lost just over three months of learning. We again present the results 
of what we would have found had we applied a similar methodology to last year’s results. 
We find that pupils made just under one month less progress in 2019/20 than pupils in 
2018/19. It is again difficult to determine the true counterfactual in this situation but, 
similarly to our argument for reading we find that: 


e Ifthe system had continued to decline at the same rate as between 2018/19 and 
2019/20 then the degree of learning loss is potentially lower — around 1 month — 
than we present here. 


e If 2019/20 was in fact a low point, then the learning loss is potentially high — again 
around 1 month. 


Figure 3.3: Estimated learning loss in mathematics for primary-aged pupils in 
2020/21 in comparison to estimates made in a similar way for outcomes in 2019/20 
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Estimates of learning loss by region and school-level disadvantage 


We have carried out some initial analysis to examine whether there are different rates of 
learning loss in different regions of England (sample sizes mean it is not possible to use 
smaller geographies) and in schools with different characteristics. We present these 
results for learning loss in reading only as the sample sizes prohibit robust analysis in 
mathematics. In reading it has been necessary to combine year groups. 


Figure 3.4 shows the mean learning loss in reading in 2020/21 by region for primary- 
aged pupils, and Figure 3.5 repeats this analysis for secondary-aged pupils. We present 
two sets of figures. The first is the simple mean of all individual pupil learning losses 
based on our modelling of outcomes in 2020/21 in comparison to 2019/20. However, this 
‘learning loss’ estimate actually reflects two features of pupil outcomes. The first - the 
element we are interested in - is the amount of learning loss that is due to the 
coronavirus (COVID-19) pandemic. The second is the different rates of progress that 
pupils typically make in each region. To control for this, we calculate the mean progress 
pupils made in each region in 2019/20 and subtract this from our estimate of learning 
loss (in other words, we assume that in the absence of coronavirus (COVID-19), pupils in 
the region would have made the same progress as pupils in the region last year). This 
provides a better estimate of the effect of the pandemic on pupil outcomes. 


We find that: 


e In terms of raw ‘learning loss’, primary-aged pupils in the North East and Yorkshire 
and the Humber lost 2.8 months, pupils in London lost the least at 1.1 months. 


e Once we adjust for historic differences in pupil progress, we find that in most 
regions (including London) primary-aged pupils have typically lost between 1.7 
and 2 months. However, pupils in the North East and Yorkshire and the Humber 
have still lost slightly more — 2.4 months and 2.2 months respectively. 


e In terms of raw ‘learning loss’, secondary-aged pupils in the North East and 
Yorkshire and the Humber lost 3.3 and 2.8 months respectively, pupils in London 
lost the least at just 0.6 months. 


e Once we adjust for historic differences in pupil progress, we find that all regions 
have experienced a loss of at least a month. Pupils in the North East and 
Yorkshire and the Humber appear to have experienced the greatest learning loss 
(2.3 and 2.4 months respectively). Learning losses were smallest in the South 
East, the East of England, and the North West. 


32 


Figure 3.4: Mean learning loss in reading for primary-aged pupils by region 


Yorkshire 
and the East East of West 
aa North East Humber NorthWest Midlands England Midlands South West South East London 
D 
” 
fe) 
0-5 


value indicate 
oO 


| | | | | : ; 
-1.6 

4.7 =a ag 21.7 ve 

-2:0 oy 2.0 

Lao: . 2PD = -2.2 

-2.4 

-2.8 -2.8 


Region 


N 
o 


Estimated Learning Loss 
, negative, 
on 


(months 
i) 
o 


' 
oo 
oO 


mLearning loss in 2020/21 m Adjusted learning loss 2020/21 


Figure 3.5: Mean learning loss in reading for secondary-aged pupils by region 
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Finally, we repeat this analysis by levels of disadvantage. In Figure 3.6 we split the mean 
learning loss in primary and secondary schools by the level of disadvantage in the school 
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attended - into high, medium and low rates of free school meal eligibility.‘* Again, we 
present a raw learning loss estimate (the outcomes of pupils in comparison to all pupils in 
2019/20) and a learning loss estimate adjusted for historic rates of progress for pupils in 
these types of schools. 


We find that: 


In terms of raw ‘learning loss’ primary-aged pupils in high-FSM schools 
experienced the greatest loss of 3.0 months and those in low-FSM schools 
experienced an apparent loss of 0.5 months. 


Once we control for historic rates of progress for pupils in different school types, 
we find that primary-aged pupils in low-FSM schools still experienced a learning 
loss (1.7 months) but this was lower than in medium and high-FSM schools (2.0 
months). 


In terms of raw ‘learning loss’ secondary aged pupils in high-FSM schools 
experienced the greatest loss of 3.7 months and those in low-FSM schools 
experienced no loss (achieving around 0.3 months higher than pupils with similar 
prior attainment in 2019/20). 


Once we control for historic rates of progress for pupils in different school types 
we find that secondary aged pupils in low-FSM schools did experience a learning 
loss (1.5 months) but this was lower than in medium and high FSM schools (2.2 
and 1.8 months respectively). 


13 For this analysis high means more than 25 per cent of pupils are eligible for free school meals, low 
means less than 10 per cent of pupils are eligible for free school meals. 
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Figure 3.6: Mean learning loss in reading for primary and secondary aged pupils by 
level of disadvantage in the school 
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These results suggest that pupils in regions and schools that typically achieve greater 
rates of progress have still experienced a learning loss. But we also find that there are 
some areas of the country that have experienced larger losses than elsewhere, and 
pupils in schools with high levels of disadvantage have lost the most. This may mean that 
there will be widening of the attainment gap between disadvantaged pupils and their 
peers. At this stage we cannot estimate the effect specifically on pupils from 
disadvantaged backgrounds and it is likely that we underestimate the effect for them 
because most pupils in high-FSM schools are not eligible for free school meals. 


We will be able to present results by pupil characteristics in our second report. 
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Conclusion 


This report represents our first assessment of the learning loss experienced by pupils in 
England as a result of the coronavirus (COVID-19) pandemic. 


Whilst there have already been several studies that have attempted to estimate learning 
loss in England the key strength of the Renaissance assessment data is that it allows us 
link outcomes in the first half of the autumn term 2020/21 with outcomes at the same 
point in 2019/20 at pupil level, and compare their progress with similar pupils in earlier 
academic years. The samples are also sufficiently large enough to allow us to see 
whether these results are consistent across different year groups. We present a method 
for estimating learning loss that is based on the expected progress for pupils based on 
their prior attainment and historic rates of progress for similar pupils. 


The degree of learning loss identified in this report is lower than has been reported by 
some other studies of pupils in England but is of a similar magnitude to those found by 
the EEF in an assessment of pupils in Year 2 and is of a similar magnitude to those 
found using Renaissance assessments in the United States. We find that all year groups 
have experienced a learning loss in reading, ranging from 1.6 months to 2.0 months. The 
learning losses in mathematics were greater, in primary schools learning losses 
averaged just over 3 months. It has not been possible to provide an estimate for 
secondary schools due to small sample sizes. '* All regions have, on average, 
experienced learning losses in reading, though the differences between regions are 
relatively small. We also find schools with high levels of disadvantage have experienced 
higher levels of loss than other schools particularly in secondary (2.2 months in schools 
with high free school meal eligibility and 1.5 months in schools with low free school meal 
eligibility). 


At this stage it is not possible to break down results by pupil characteristics, or to model 
the progress typically made in Star Assessments by different pupil groups, but this will be 
possible when the data is matched with information held in the National Pupil Database 
and will be included in our second report. We will develop our models to account for 
these different rates of progress made by different pupil groups. 


That data will also enable us to assess the impact of the coronavirus (COVID-19) 
pandemic on the gap in attainment between disadvantaged pupils and their peers. 
Consistent with other studies, the initial analysis presented in this report suggests that 
that gap may have widened as a result of the pandemic. 


14 All of these estimates are based on an assumption that, in the absence of the coronavirus (COVID-19) 

pandemic, pupils would progress at the same rate as pupils in the past. It is of course possible that pupils 
this year would have progressed at a different rate, so we illustrate this in the report by applying the same 
approach to predicting last year’s results and find smaller fluctuations than the losses estimated here. 
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Annex 


Data 


Renaissance Learning provided a dataset to the Education Policy Institute that comprised 
all assessments undertaken in schools in England between 1% August 2017 and 25" 
October 2021. A total of 11,685,315 records were provided (one per assessment). The 
data consisted of the following variables: 

e a‘meaningless’ pupil identifier’ 

e aschool URN where available 

e subject (reading and mathematics) 

e academic year 

e year group of pupil 

e date of test 

e scaled score 

e percentile rank 


The school region and percentage of pupils eligible for free school meals was taken from 
an extract from Get Information About Schools taken on 16 November 2020. This was 
linked to the pupil data using the school URN. Note that the URN data was complete for 
around 90 per cent of pupils. 


Number of assessments 


Table A1: The number of assessments undertaken in Star Reading and Star Maths 
by half-term 2017/18 


Half-term | Reading | Mathematics 


Autumn 1 | 807,000 32,000 
Autumn 2 | 496,000 33,000 
Spring 1 519,000 24,000 
Spring 2 442,000 30,000 
Summer 1 | 447,000 22,000 
Summer 2 | 619,000 35,000 


15 The identifier was consistent within the dataset to allow linking of different assessment records but has 
no real world meaning 
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Table A2: The number of assessments undertaken in Star Reading and Star Maths 


by half-term 2018/19 


Table A3: The number of assessments undertaken in Star Reading and Star Maths 
by half-term 2019/20 


Table A4: The number of assessments undertaken in Star Reading and Star Maths 


by half-term 2020/21 


Half-term | Reading | Mathematics 
Autumn 1 | 937,000 45,000 
Autumn 2 | 578,000 44,000 
Spring 1 615,000 31,000 
Spring 2 561,000 41,000 
Summer 1 | 423,000 22,000 
Summer 2 | 717,000 44,000 


Half-term | Reading | Mathematics 
Autumn 1 | 1,119,000 | 55,000 
Autumn 2 | 631,000 51,000 
Spring 1 697,000 33,000 
Spring 2 325,000 27,000 
Summer 1 | 32,000 1,000 
Summer 2 | 49,000 4,000 


Half-term 


Reading 


Mathematics 


Autumn 1 


1,039,000 
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58,000 


Regression analysis '° 


Table A5: Regression coefficients and R-squared value for the estimated learning 
loss in 2020/21 (modelled on 2019/20 data) — reading 


Year | Constant | Coefficient — prior attainment | R-squared 
Year 3 | 147.885 | 0.957 0.475 
Year 4 | 140.224 | 0.925 0.563 
Year 5 | 131.479 | 0.948 0.590 
Year 6 | 110.491 | 1.006 0.600 
Year 7 | 133.272 | 0.946 0.566 
Year 8 | 110.471 | 0.982 0.576 
Year9 | 146.774 | 0.919 0.610 


Table A6: Regression coefficients and R-squared value for the estimated learning 
loss in 2020/21 (modelled on 2019/20 data) — mathematics 


Year Constant | Coefficient — prior attainment | R-squared 


Years 3-7 | 172.183 | 0.835 0.637 


Table A7: Regression coefficients and R-squared value for the estimated learning 
loss in 2019/20 (modelled on 2018/19 data) — reading 


Year | Constant | Coefficient — prior attainment | R-squared 
Year 3 | 138.936 | 0.962 0.485 
Year 4 | 132.543 | 0.927 0.568 
Year 5 | 124.094 | 0.959 0.600 
Year 6 | 98.516 1.024 0.605 
Year 7 | 108.566 | 0.980 0.601 
Year 8 | 100.921 | 0.989 0.582 
Year 9 | 123.805 | 0.941 0.621 


Table A8: Regression coefficients and R-squared value for the estimated learning 
loss in 2019/20 (modelled on 2018/19 data) — mathematics 


16 Note that year 7 is grouped with primary since we are primarily interested in learning loss that occurred in 
the summer term when this group were in year 6. 
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Year 


Constant 


Coefficient — prior attainment 


R-squared 


Years 3-7 


161.530 


0.866 


0.670 


Days between assessments 


Table A9: Percentiles of the number of days between tests taken in the first half of 
the autumn term for reading — 2018/19 to 2019/20 


Year | 5‘ percentile | 25" percentile | 50‘ percentile | 75‘ percentile | 95" percentile 
Year 3 | 328 355 364 372 408 
Year 4 | 328 357 364 373 410 
Year 5 | 328 358 364 372 409 
Year 6 | 330 358 364 371 407 
Year 7 | 333 358 366 375 395 
Year 8 | 341 356 363 371 390 
Year 9 | 344 359 365 374 393 


Table A10: Percentiles of the number of days between tests taken in the first half of 
the autumn term for reading — 2019/20 to 2020/21 


Year | 5‘ percentile | 25" percentile | 50‘ percentile | 75‘ percentile | 95" percentile 
Year 3 | 322 349 364 373 402 
Year 4 | 320 351 365 373 400 
Year 5 | 319 303 365 373 399 
Year6 | 319 353 365 373 401 
Year 7 | 330 359 369 378 397 
Year 8 | 342 361 371 380 395 
Year 9 | 342 362 371 383 399 


Table A11: Percentiles of the number of days between tests taken in the first half of 


the autumn term for mathematics — 2018/19 to 2019/20 


Year | 5‘ percentile | 25‘ percentile | 50‘ percentile | 75‘ percentile | 95" percentile 
Year 3 | 332 349 363 371 384 
Year 4 | 332 356 364 371 403 
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Year 5 | 331 353 364 372 401 
Year 6 | 329 347 362 371 403 
Year 7 | 334 337 343 358 392 
Year 8 | 336 356 360 371 393 
Year 9 | 345 361 376 379 386 


Table A12: Percentiles of the number of days between tests taken in the first half of 


the autumn term for mathematics — 2019/20 to 2020/21 


Year | 5‘ percentile | 25" percentile | 50‘ percentile | 75‘ percentile | 95" percentile 
Year 3 | 334 361 367 373 390 
Year 4 | 333 354 365 373 391 
Year 5 | 335 358 367 375 398 
Year 6 | 330 358 365 373 393 
Year 7 | 356 359 363 365 378 
Year 8 | 340 359 363 368 389 
Year 9 | 343 364 367 393 398 
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